4 research outputs found
ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes
Understanding the continuous states of objects is essential for task learning
and planning in the real world. However, most existing task learning benchmarks
assume discrete(e.g., binary) object goal states, which poses challenges for
the learning of complex tasks and transferring learned policy from simulated
environments to the real world. Furthermore, state discretization limits a
robot's ability to follow human instructions based on the grounding of actions
and states. To tackle these challenges, we present ARNOLD, a benchmark that
evaluates language-grounded task learning with continuous states in realistic
3D scenes. ARNOLD is comprised of 8 language-conditioned tasks that involve
understanding object states and learning policies for continuous goals. To
promote language-instructed learning, we provide expert demonstrations with
template-generated language descriptions. We assess task performance by
utilizing the latest language-conditioned policy learning models. Our results
indicate that current models for language-conditioned manipulations continue to
experience significant challenges in novel goal-state generalizations, scene
generalizations, and object generalizations. These findings highlight the need
to develop new algorithms that address this gap and underscore the potential
for further research in this area. See our project page at:
https://arnold-benchmark.github.ioComment: The first two authors contributed equally; 20 pages; 17 figures;
project availalbe: https://arnold-benchmark.github.io
NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities
We present Neural Signal Operated Intelligent Robots (NOIR), a
general-purpose, intelligent brain-robot interface system that enables humans
to command robots to perform everyday activities through brain signals. Through
this interface, humans communicate their intended objects of interest and
actions to the robots using electroencephalography (EEG). Our novel system
demonstrates success in an expansive array of 20 challenging, everyday
household activities, including cooking, cleaning, personal care, and
entertainment. The effectiveness of the system is improved by its synergistic
integration of robot learning algorithms, allowing for NOIR to adapt to
individual users and predict their intentions. Our work enhances the way humans
interact with robots, replacing traditional channels of interaction with
direct, neural communication. Project website: https://noir-corl.github.io/